In  1999,  the  North  Atlantic  Treaty  Organization  issued  a  technical  report  lay¬ 
ing  out  a  “Code  of  Best  Practice”  for  command  and  control  assessment.1  Al¬ 
though  specifically  aimed  at  command  and  control,  this  document  offers  a 
framework  for  thinking  about  the  changing  nature  of  war  gaming.  In  the 
opinion  of  numerous  practitioners  and  observers,  war  gaming  has  reached  a 
turning  point:  the  changing  basis  of  international  security  at  the  dawn  of  the 
twenty- first  century  makes  gaming  an  especially  valuable  tool,  but  a  fundamen¬ 
tal  reformation  of  gaming  is  required  for  it  to  achieve  its  potential. 

The  Code  of  Best  Practice,  as  a  unifying  and  overarching  framework,  allows 
us  to  take  stock  of  the  present  state  of  war  gaming,  to  highlight  the  primary  chal¬ 
lenges  that  the  war-gaming  community  faces,  and  to  propose  steps  to  improve 
every  aspect  of  war  gaming.  It  makes  four  central  points. 

First,  as  shown  in  figure  1,  the  framework  of  a  good  war  game  should  be 
broadly  based  on  the  principles  of  sound  operational  analysis.  Thus  the  corner¬ 
stone  of  any  game  must  be  a  clear  and  unambiguous  formulation  of  the  problem 
to  be  addressed — the  reason  the  game  is  to  be  played.  A  game’s  sponsors  need  to 
articulate  very  clearly  the  real  issues  of  interest  so  that  designers  may  develop 
(for  the  sponsors’  approval)  a  conceptual  framework  within  which  these  issues 
can  be  suitably  analyzed. 

Second,  as  the  Nato  document  instructs,  the  game’s  designers  should  identify 
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and  address  organizational  and  cultural  issues  that 
emerge  from  the  conceptual  framework.  What  as¬ 
sumptions  are  to  be  accepted,  for  example,  about 
the  values,  behavior,  and  decision  processes  of  the 
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various  players? 
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Third,  the  war  game  must  devise  relevant  scenarios.  As  is  widely  understood, 
no  single  scenario  is  adequate  for  the  full  range  of  issues  found  in  a  major  game. 
Sponsors  should  expect  to  be  presented  with  “families”  of  scenarios;  a  systematic 


FIGURE  1 

WAR-GAMING  FRAMEWORK 


and  efficient  mechanism 
to  generate  appropriate  al¬ 
ternative  scenarios  allows 


a  game  to  focus  on  the 
most  interesting  aspects  of 
the  problem  being  studied. 

Fourth,  Nato’s  Code  of 
Best  Practice  envisions 
the  use  of  “measures  of 
merit”  to  draw  out  insights 
about  the  game’s  results; 
for  contemporary  scenar¬ 
ios,  hierarchies  of  interre¬ 
lated — and,  increasingly, 
nontraditional — measures 
are  necessary.  These  mea¬ 
sures,  in  turn,  require  the 
collection  of  appropriate  data  and  the  application  of  suitable  analytical  tools  to 
be  useful.  For  instance,  ancillary  tools  can  be  used  to  perform  analyses  before  a 
game  (perhaps  to  define  fruitful  parts  of  “scenario  space”),  during  it  (to  assess 
“moves”),  and  after  it,  especially  to  relate  outcomes  to  measures  of  merit.  The 
Nato  Code  considers  it  vital  to  perform  risk  analyses  to  illuminate  the  uncertain¬ 
ties  associated  with  the  issues  of  interest  to  the  sponsor;  many  a  game  partici¬ 
pant  (and  sponsor)  has  drawn  a  misleading  inference  from  the  idiosyncratic 
outcome  of  a  single  game.  Finally,  the  results  of  the  assessments  must  be  doc¬ 
umented,  so  there  can  be  both  peer  reviews  and  a  foundation  upon  which  future 
analyses  can  be  built. 

Although  figure  1  does  not  formally  specify  it,  the  Code  of  Best  Practice  em¬ 
phasizes  that  an  extensive  feedback  arrangement  is  needed  to  share  insights 
among  individuals  carrying  out  these  successive  processes  as  game  planning 
progresses.  Further,  the  overall  team  must  be  an  interdisciplinary  one — com¬ 
prising  operations  analysts,  war-game  designers,  experimental  designers,  com¬ 
puter  scientists,  social  scientists,  and  so  on — if  it  is  to  address  all  of  the  issues  of 
concern  to  a  sponsor.  A  “good  game,”  then,  blends  clear  problem  formula¬ 
tion,  technical  virtuosity,  accurate  data,  scenario  creativity,  appropriate  decision 
rules,  and  credible  evaluation  procedures.  The  rest  of  this  article  treats  some  of 
these  points  in  greater  depth. 
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PROBLEM  FORMULATION 

There  is  no  shortage  of  problems  amenable  to  useful  analysis  by  war-gaming 
techniques.  In  fact,  policy  makers  are  likely  to  find  war  gaming  the  most  effective 
tool  for  clarifying  many  issues  and  sets  of  issues  that  can  be  expected  to  come  to 
the  foreground  in  the  near  and  middle  term.  Let  us  here  consider  a  few  problems 
for  which  war  gaming  could  be  particularly  appropriate. 

Strategic  Visions.  An  indication  of  the  variety  of  problems  to  which  gaming 
might  be  applied  as  an  analytical  tool  is  the  set  of  three  lists  of  strategic  prob¬ 
lems  assembled  recently  by  former  secretary  of  defense  William  J.  Perry  and  for¬ 
mer  assistant  secretary  of  defense  Ashton  B.  Carter,  in  their  book  Preventive 
Defense.2  Their  “A  list”  comprises  potential  (and  possibly  preventable)  future 

-  matters  that  could  threaten  the 

The  overall  team  must  be  an  interdisciplinary  survival,  way  of  life,  and  position 

One - Comprising  Operations  analysts,  in  the  world  of  the  United  States 

war-game  designers,  experimental  designers,  (such  as  a  resurgent  and  hostile 
computer  scientists,  social  scientists,  and  SO  on.  Russia,  uncontrolled  prolifera- 
-  tion  of  weapons  of  mass  destruc¬ 
tion,  or  catastrophic  terrorism).  The  “B  list”  contains  direct  threats  (deterrable 
through  ready  forces)  to  vital  American  interests  (for  instance,  major  theater 
wars).  The  “C  list”  cites  problems  (like  Kosovo,  Bosnia,  and  Somalia)  that  “indi¬ 
rectly  affect  U.S.  security  but  do  not  directly  threaten  U.S.  interests.”3  Many  of 
these  issues,  particularly  those  on  the  “A  list,”  have  yet  to  be  explored  adequately 
in  war  games. 

Homeland  Defense.  In  its  recent  report,  “Seeking  a  National  Strategy:  A  Concept 
for  Preserving  Security  and  Promoting  Freedom,”  the  congressionally  estab¬ 
lished  Hart-Rudman  Commission  emphasizes  the  need  to  enhance  what  it  calls 
“homeland  security”  to  deal  with  emerging  world  threats.4  A  third  and  final 
phase  of  that  study  will  address  a  variety  of  associated  questions:  Are  responsi¬ 
bilities,  authorities,  and  accountabilities  clear?  Do  integrating  mechanisms  ex¬ 
ist?  What  capabilities  will  be  needed?  Is  the  overall  capacity  sufficient,  and  if  so, 
will  it  continue  to  be?5  A  suitably  designed  set  of  war  games  would  be  a  promis¬ 
ing  way  to  illuminate  these  issues.6 

Operational  Tempo.  One  of  the  driving  issues  in  the  1997  Quadrennial  Defense 
Review  was  the  necessity  to  devise  “architectures”  and  personnel  policies  to  al¬ 
low  U.S.  forces  to  respond  to  operational  demands  that  were  expected  to  be  high 
enough  to  put  pressure  on  unit  training  and  maintenance,  as  well  as  morale  and 
retention.  The  “Dynamic  Commitment”  war  game  was  developed  and  played  to 
address  that  issue;  it  is  being  revised  to  serve  the  same  need  for  the  Quadrennial 
Defense  Review  of  2001. 7  That  game — which  is  to  play  a  single  scenario,  drawn 
randomly  from  a  list  of  sixty-one  “vignettes” — is  itself  a  case  in  point,  showing 
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that  contemporary  gaming  does  not  reflect  sufficient  understanding  of  risks  and 
uncertainties.  The  consequences  of  a  particular  vignette  being  played  out  in  a 
single  game,  and  of  drawing  conclusions  therefrom  about  levels  of  demands  that 
can  be  placed  on  U.S.  equipment  and  personnel,  are  worrisome. 

Service  Transformation.  Each  of  the  military  services  is  in  the  midst  of  sweeping 
modernization  designed  to  take  advantage  of  opportunities  offered  by  the  infor¬ 
mation  age.  Specific  initiatives  include  the  Navy’s  network-centric  warfare,  the 
Army’s  “Future  Combat  System  for  Smaller  Scale  Contingencies,”  the  air  expedi- 

-  tionary  forces  of  the  Air  Force, 

How  can  scenarios  be  kept  (in  Albert  Einsteins  and  “Operational  Maneuver  from 
formulation )  as  simple  as  necessary —  the  Sea”  of  the  Marine  Corps.  War 

but  no  simpler?  games  have  contributed  to  pre- 

-  liminary  assessments  of  each  of 

these  concepts  singly,  but  there  has  been  no  attempt  to  game  the  totality  of  their 
effects.  Doing  so  would  appear  to  be  a  high-priority  matter. 

Joint  Vision  2020.  The  chairman  of  the  Joint  Chiefs  of  Staff,  in  the  recent  doc¬ 
trinal  white  paper  Joint  Vision  2020,  conceives  “a  joint  force  capable  of  full  spec¬ 
trum  dominance,  persuasive  in  peace,  decisive  in  war,  and  preeminent  in  any 
form  of  conflict.”8  The  document  reaffirms  as  the  prerequisite  of  full-spectrum 
dominance  four  operational  concepts — dominant  maneuver,  precision  engage¬ 
ment,  focused  logistics,  and  full-dimensional  protection — identified  in  an  ear¬ 
lier  white  paper,  Joint  Vision  2010.  These  four  operational  concepts  in  turn 
depend  on  three  factors:  interoperability  (joint  force,  interagency,  and  multina¬ 
tional),  innovation  leading  to  transformation,  and  “decision  superiority”  (to  al¬ 
low  commanders  to  “make  better  and  faster  decisions  than  their  opponents”). 
All  of  these  factors,  as  well  as  their  relationship  to  the  central  operational  con¬ 
cepts  of  Joint  Vision  2020,  are  very  attractive  subjects  for  gaming. 

ORGANIZATION  AND  CULTURE 

In  games  played  by  coalition  allies  prior  to  Operation  Desert  Storm,  differences 
in  cultures  were  sometimes  recognized  as  a  major  factor.  For  instance,  British 
analyses  reflected  a  particular  appreciation  of  Iraqi  characteristics  that  pro¬ 
foundly  affected  the  planning  and  operational  concepts  of  the  British  forces  in 
the  theater.9 

Cultural  differences  were  again  acknowledged  as  central  strategic  factors  in 
1999,  during  Nato’s  coercive  air  campaign  to  terminate  internecine  hostilities 
in  Kosovo.  The  subsequent  debate  about  what  actually  prompted  Slobodan 
Milosevic’s  acquiescence  to  Nato’s  demands  has  produced  at  least  one  analysis  of 
the  cultural  and  political  dynamics  of  the  Serbian  leadership.10 
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It  has  been  widely  appreciated  that  war  games  require  a  much  better  theoreti¬ 
cal  basis  than  is  now  available  for  treating  these  matters  in  future  conflict  situ¬ 
ations.  One  potential  source  of  some  necessary  insight  may  emerge  from  work 
being  done  in  the  Office  of  Naval  Research  on  “Adaptive  Architectures  for  Com¬ 
mand  and  Control.”  It  examines  the  command-and-control  staffs  of  various  na¬ 
tions  for  pertinent  “cultural  artifacts”  and  their  potential  influence  on  decision 
making.  In  addition,  the  war-gaming  community  would  do  well  to  draw  on  the 
efforts  of  sociologists  and  political  scientists,  who  could  analyze  the  underlying 
cultural  forces  at  work  in  such  recent  operations  as  Somalia  and  Kosovo,  and 
who  can  be  consulted  in  planning  games  for  prospective  involvement  in  foresee¬ 
able  crises. 

SCENARIOS 

Today,  basic  issues  in  the  selection  and  development  of  scenarios  are  being  ex¬ 
amined.  Can  a  baseline  scenario  be  used  for  a  series  (or  “cluster”)  of  games? 
How  can  scenarios  be  kept  (in  Albert  Einstein’s  formulation)  as  simple  as  neces¬ 
sary — but  no  simpler?  Can  “excursions”  into  important  issues  be  accommo¬ 
dated,  and  if  so,  in  what  ways? 

Clearly,  no  simple  answers  to  these  questions  exist,  but  there  is  a  fundamental 
principle  that  game  designers  today  should  acknowledge — that  no  single  sce¬ 
nario  can  adequately  illuminate  risk  and  uncertainty.  The  challenge  is  to  develop 
an  efficient  mechanism  for  finding  and  exploring  regions  of  “scenario  space” 
where  key  factors  play  in  significant  ways.  The  Nato  Code  of  Best  Practice  offers 
one  approach  to  the  problem,  a  scenario  framework  that  subsumes  three  major 
categories — external  factors  (the  political,  military,  and  cultural  situation),  the 
capabilities  of  actors  (friendly  and  adversary  forces,  noncombatants),  and  the 
environment  (geography,  terrain,  and  weather). 

As  an  illustration  of  how  such  a  framework  might  be  used  to  develop  a  base¬ 
line  scenario  (and  possibly  scenario  excursions),  consider  a  methodology  that 
enumerates  the  factors  applicable  to  a  given  game.11  For  each  of  those  factors,  a 
number  of  values  (specific  geographies,  particular  orders  of  battle,  etc.)  can  be 
assigned,  each  making  a  scenario  more  or  less  challenging  in  some  respect  that  is 
significant  in  terms  of  a  game’s  objectives.  Between  the  bounding  (“easy,”  “very 
difficult”)  values  for  each  factor  lie  the  elements  of  a  potentially  interesting  base¬ 
line  scenario;  alternative  scenarios  can  be  readily  produced  for  sensitivity  analy¬ 
ses  by  selecting  different  values  for  particular  factors.  In  effect,  this  approach 
generates  a  very  large  experimental-design  matrix,  each  cell  of  which  corre¬ 
sponds  to  a  specific  scenario.  In  traditional  scientific  experimentation,  a  num¬ 
ber  of  iterations  would  be  run  for  selected  matrix  cells  in  order  to  achieve 
statistically  meaningful  results;  statistical  uncertainty  would  be  a  function  of 
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the  number  of  cells  examined  and  the  number  of  independent  trials  of  each.  In 
war  games,  of  course,  a  “full  factorial  experiment”  would  be  impossible;  still,  it 
would  be  prudent  to  play  at  least  a  sampling  of  variants — a  “sparse,  fractional 
factorial  experiment.” 

Closely  related  to  scenarios  is  consideration  of  risk  and  uncertainty.  As  the 
Code  notes,  a  useful  way  to  display  and  characterize  areas  of  uncertainty  in  a 

-  game  is  to  play  variations  of  the 

Today’s  State- of -practice  technology  simply  scenario.  In  doing  so,  however,  it 
collects  Stand-alone  collaboration  tools —  is  important  to  take  account  of, 

however,  the  state  of  the  art  has  advanced  to  and  offset,  the  effects  of  learning 

the  point  of  integrating  those  capabilities  into  that  occurs  in  the  play  of  a  game. 

“ virtual  buildings.”  For  instance,  the  sequence  of  vari- 

-  ations  should  anticipate  and  min¬ 
imize  the  “carry  forward”  insights  obtained  in  each  variation;  one  way  to  do  this 
is  to  make  the  new  problem  appear  different  to  the  participants  but  have  it  con¬ 
tain  the  same  essential  stimuli.  It  will  almost  certainly  not  be  possible  to  run 
enough  iterations  to  bound  measures  of  merit  as  tightly  as  a  physical  scientist 
would  wish;  nevertheless,  to  some  extent  well  designed  pre-  and  postgame  anal¬ 
yses  can  refine  those  estimates. 

A  more  basic  issue  is  the  estimation  of  risk.  Risk  analysis  as  a  discipline  is  well 
developed  in  a  number  of  fields,  such  as  the  insurance  industry  and  stock  bro¬ 
kerages,  but  in  the  context  of  national  security  there  is  little  agreement  even 
about  the  definition  of  risk  itself.  This  is  becoming  a  pressing  issue,  because  the 
congressional  mandate  of  the  Quadrennial  Defense  Review  specifically  requires 
“a  comprehensive  discussion  of  [the]  national  defense  strategy  of  the  United 
States  and  the  force  structure  best  suited  to  implement  that  strategy  at  a  low  to 
moderate  level  of  risk.”12  To  meet  this  requirement  the  national  security  com¬ 
munity  will  need  to  agree  on  definitions  of  risk,  definitions  that  are  amenable  to 
evaluation  in  future  war  games. 

MEASURES  OF  MERIT 

For  decades  gamers  have  employed  the  familiar  operations-analysis  device  of 
“measures  of  effectiveness”  to  structure  game  outcomes  and  relate  them  to 
sponsors’  concerns.  In  recent  years,  however,  the  concept  of  measures  of  effec¬ 
tiveness  has  been  broadened,  resulting  in  the  idea  of  “measures  of  merit.”13  As 
discussed  in  the  Nato  document,  this  conception  not  only  embraces  the  conven¬ 
tional  measures  of  effectiveness  but  allows  a  linked  hierarchy  of  increasingly 
specific  metrics  to  be  considered  as  well.  For  example,  the  evaluation  measures 
of  a  game  might  employ  measures  each  of  which  “nests”  within  the  next  to  pro¬ 
vide  both  broad  and  detailed  attention  as  appropriate.  An  example  follows: 
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•  Measures  of  policy  effectiveness,  assessing  the  extent  to  which  the 
participants  in  an  operation  are  able  to  achieve  national  or  international 
security  objectives; 

•  Measures  of  force  effectiveness,  examining  the  purely  military  effectiveness  of 
a  force  in  terms  of  its  primary  task  (such  as  the  time  required  to  halt  an 
attack); 

•  Measures  of  mission  effectiveness,  appraising  the  ability  of  the  military  force 
to  perform  key  subordinate  or  subsidiary  missions; 

•  Measures  of  functional  performance,  evaluating  the  success  of  a  particular 
weapon  system  or  command-and-control  organization  in  important  tasks, 
such  as  target  engagement; 

•  Dimensional  parameters,  the  properties  or  characteristics  (such  as 
bandwidth  and  resistance  to  jamming)  of  a  specific  system,  such  as  a 
communications  network. 

Game  designers  might  usefully  devise  measures  for  each  level  of  this  hierar¬ 
chy,  and  analysts  might  explore  their  relationships  during  the  course  of  the 
game.  At  the  lower  end  of  the  hierarchy,  extensive  analyses  have  been  performed 
for  traditional  warfare;  that  literature  is  being  expanded  upon  to  embrace 
information  superiority.14  It  would  be  necessary,  however,  to  formulate  mean¬ 
ingful  measures  of  merit  for  the  top  of  the  hierarchy.  In  one  promising  effort  in 
this  direction,  economic  measures  were  used  to  reflect  the  societal  impact  of 
military  operations.15  Participants  were  asked  to  estimate  the  effect  that  postu¬ 
lated  crises  might  have  on  such  indicators  as  the  Dow  Jones  Industrial  Average, 
the  price  of  a  barrel  of  crude  oil,  or  the  exchange  rate  between  the  dollar  and  the 
deutsche  mark. 

As  the  Nato  Code  of  Best  Practice  concludes,  games  are  not  suitable  for  every 
analytical  question.  Indeed,  no  single  assessment  technique  is  likely  to  be  suffi¬ 
cient  (see  table  1 ).  Since  games  are  increasingly  likely  to  address  such  concepts  as 
information  superiority  and  information  dominance,  assessment  tools  must  ac¬ 
count  for  both  friendly  and  adversary  information  processes.  In  addition,  disci¬ 
pline  is  necessary;  formal  experimental-design  matrices  may  be  advisable,  or 
multiple  iterations  of  increasingly  fine-grained  analytical  routines  may  have  to 
be  done  (for  instance,  in  successive  attempts  before  a  game  to  identify  fruitful 
aspects  of  the  scenario  environment,  clarify  assumptions,  assign  values  for  key 
parameters,  and  model  details). 

Newly  developed  sophisticated  collaboration  tools  may  revolutionize  war 
games  by  allowing  geographically  dispersed  individuals  to  participate  fully  in 
deliberations  and  decisions.  Today’s  state-of-practice  technology  simply 
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collects  stand-alone  collaboration  tools — like  video  teleconferencing,  shared 
whiteboards,  and  Internet  chat  rooms.  However,  the  state  of  the  art  has  ad¬ 
vanced  to  the  point  of  integrating  those  capabilities  into  “virtual  buildings”  in 
which  participants  interact  in  real  time.  Efforts  are  under  way  to  improve 
“scalability”  (usefulness  for  various  numbers  of  players  and  complexities  of  sce¬ 
nario)  and  to  deal  with  security  issues  regarding  the  transmission  of  game  data. 


TABLE  1 

SPECTRUM  OF  CANDIDATE  ASSESSMENT  TECHNIQUES 


Techniques 

Typical 

Application 

Systems 

People 

Ops/ 

Mission 

Resources 

Lead  Time 

Credibility 

Create 

Use 

Analysis 

Closed  Form; 

Statistical 

Analytical 

Assumed  or 

Simulated 

Simulated 

Relatively 

Modest 

Weeks  to 

Months 

Weeks  to 

Months 

Fair  to 

Moderate 

Constructive 

Force  on  Force 

Models; 

Communication 

Systems 

Simulated 

Assumed  or 

Simulated 

Simulated 

Moderate 

Months  to 

Years 

Weeks  to 

Months 

Moderate 

Virtual 

War  games; 
Human  in 
the  Loop 

Simulated 

Real 

Simulated 

High 

Months  to 

Years 

Weeks  to 

Months 

Moderate  to 
High 

Live 

CPX* 

FTX** 

Real 

Real 

Real  or 

Simulated 

Very  High 

Years 

Weeks  to 

Months 

High 

Actual  Ops 

After  Action  Reports; 
Lessons  Learned 

Real 

Real 

Real 

Extremely 

High 

N/A 

N/A 

Very  High 

*CPX-Command  Post  Exercise 
**FTX-Field  Training  Exercise 

One  of  the  major  advantages  that  these  emerging  collaborative  gaming  tools 
offer  is  the  possibility  that  principals — commanders,  heads  of  agencies,  senior 
executives — will  be  able  to  participate  personally.  The  demands  on  the  time  of 
such  individuals  normally  make  it  difficult  for  them  to  get  involved  in  war 
games,  especially  if  travel  is  involved;  typically  they  must  delegate  such  matters 
to  subordinates.  Distributed,  collaborative  war-gaming  technologies  will  make 
it  possible  for  actual  decision  makers  to  play,  increasing  both  the  fidelity  of  the 
games  and  the  real  value  of  the  entire  activity  by  educating  the  principals  di¬ 
rectly  about  the  intricacies  and  nuances  of  the  problems  being  considered. 
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